Current Topics in Artificial Intelligence: Regularization

This short survey discusses the role of regularization in current deep learning research. Up till now, dropout remains the most popular choice of all deep neural network regularization techniques, and is what this survey is centered around. We first give a general introduction and interpretation of dropout (Section 1), followed by some follow-up works which either improve speed or offer analysis (Section 2). Then methods other than dropout, namely DropConnect and Batch Normalization, are introduced (Section 3) to provide comparison and contrast. Finally the necessity of regularization is argued (Section 4). This is the second of the four short surveys. 1 Dropout Regularization is important for deep neural networks, because they usually have millions of parameters to learn, and given the limited training data, overfitting is quite likely to happen. Dropout [1] stood out as a simple yet effective regularization technique, and first attracted major attention when it greatly alleviated the overfitting problem of AlexNet [2], the ILSVRC 2012 winner. 1.1 Formulation and Intuition The idea of dropout is quite simple. It introduces stochasticity into the neural network by (temporarily) dropping random units out, along with all its incoming and outgoing connections. This is equivalent to sampling a “thinned” network from the complete architecture. Concretely: • During each training iteration, drop each unit in a layer out with probability p (usually set to 0.5, which means randomly dropping half of the units in the current layer). • During testing, multiply all the outgoing weights of the units by p and make final prediction using all the units. The intuition behind dropout is as follows. If the capacity of the network is much larger than the data it is representing, then overfitting is likely to happen. In terms of weights, this could mean that a large number of weights have learned to “cooperate” or “co-adapt” to fit the data point. These complex co-adaptations could go wrong on the new test data. On the other hand, by randomly dropping half of the units, a unit is forced to work with different “co-workers” during each iteration of training. As a result, it is likely to learn something that is individually useful. But after learning is done, we no longer want to use the “thinned” networks and would like to utilize all the knowledge that the network has learned. Since the weights of the next layer is learned when we drop out half of the units in the current layer, if we fix the outgoing weights of all the units, then